There are quite some activities to do in Walvis Bay.
Dolphin watching.
Checkout mola mola cruises for dolphin and seal cruises tel +264 64 205 511.
Kite Surfing.
Head on down to the lagoon to ride the waves. lessons and euipment by (http://www.namibiakite.com))
Go fishing.
Rods can be rented. you need a fishing permit though which can be bought at the ministry of fisheries for 14 n$ for 30 days inspectorate office, 1st street, phone: +264 64 2016111, monday-sunday 8-13. speak to the locals about the various fishing spots.
Dune boarding.
Buy a dune board in town and head 6km out of town to dune 7 for an inexpensive thrill.